1. Field of the Invention
The invention relates to a technique, specifically apparatus and accompanying methods, for implementing an on-demand "Tool Glass" based desktop user interface. In particular, by sensing whether a user is explicitly touching an input pointing device, a Tool Glass sheet is automatically either displayed or dismissed preferably through a controlled fade in/fade out operation. This technique is particularly, though not exclusively, suited for use in conjunction with such an interface that accepts two-handed user input. Furthermore and advantageously, through the present invention, touch sensing can readily be used to provide "on-demand" display and dismissal, again with preferably controlled fading, of substantially any display widget, e.g., a toolbar, based on sensed contact between a hand of a user and a corresponding touch sensitive input device.
2. Description of the Prior Art
A continuing challenge in the field of computation is to develop an interface that, to the extent possible, facilitates and simplifies interaction between a user and a computer and thus enhances an experience which that user then has with his(her) computer, i.e., a so-called "user experience".
Over the past several years or so, a computer mouse and a keyboard collectively operating with a graphical user interface have become a rather ubiquitous user interface (UI). Through such a UI, a user seeing a graphical display, such as an operating system desktop or a window of an application, positions a cursor on a display screen by directly moving the mouse in two dimensions across a suitable surface. Movement of the cursor simply mimics the movement of the mouse. Various buttons are located on the top of the mouse to enable the user to cause a mouse "click" (button depression) whenever (s)he appropriately positions the cursor at a desired location on the display, such as, e.g.: over a desired icon in a toolbar; over a selection in a pull-down menu; or, within an application window itself, over a selected position in a document. Appropriate software, in, e.g., an application, interprets the "click", in a contextual setting, governed by the cursor location and a then current state of the application, as a particular command and then suitably performs that command.
With this interface, a preferred (dominant) hand of a user, typically a right hand for right-handed individuals, manipulates the mouse, while a non-preferred (non-dominant) hand, a left hand for these individuals, may either manipulate the keyboard or not. Commands can be entered either through the mouse or the keyboard, with the particular use of either device being governed by the current state of, e.g., the application. Through such an interface, the use of the keyboard and mouse are staggered in time. The user manipulates one device, often with one hand, and then manipulates the other, often with the same and/or different hand, but does not manipulate both devices at the same time. Hence, such a conventional interface is commonly referred to as being "one-handed".
Unfortunately, the interaction afforded by a conventional one-handed (keyboard-mouse) UI has simply not kept pace with the tasks which many computer users seek to perform through that interface. In essence, a practical limit has been reached as to the complexity of tasks which a user can readily accomplish through such an interface.
Specifically, as users seek to perform increasingly sophisticated tasks through an application program, they are becoming increasingly frustrated owing to the practical limitations inherent in a conventional one-handed UI. In that regard, a significant number of mouse clicks and/or other mouse and keyboard manipulations is often required to accomplish various tasks through that conventional UI. This, in turn, can impose an cognitive burden on a user, which, for repetitive operations, can be appreciable and rather fatiguing. In particular, conventional graphical user interfaces often position command menus, icons and other user-actuable ("clickable") visual objects along one or more edges on a display screen and peripherally located to a centrally displayed application area. Often, these icons and objects are organized into one or more toolbars and/or other visual groupings (the visual objects, icons, toolbars and other groupings are all commonly referred to as UI "widgets"). Frequently, the program permits the user to appropriately set a software switch(es), through, e.g., a dialog box of "option" settings, that explicitly displays or dismisses any or all of the toolbars and other groupings in an attempt to reduce screen clutter. By dismissing such widgets, added display space can be allocated to displaying application information in lieu of widgets.
However, given the peripheral location of the widgets on the display screen, then, to invoke a desired operation, the user must generally direct his(her) focus of attention back and forth between the application area on the screen and the peripheral "widget" area and correspondingly move the mouse between the two to separately select command(s) and operands. Disadvantageously, this constant shift of attention mentally tires the user as (s)he is forced to repetitively "re-acquire" his(her) current context, and these mouse movements increase user task time; hence collectively depreciating the "user experience". These drawbacks are exacerbated as the display size increases for a constant display resolution.
In an effort to circumvent these drawbacks inherent in a conventional one-handed UI, the art teaches the use of two-handed UI and particularly one that manipulates a so-called "Tool Glass" widget (which, for simplicity, will simply be referred to as a "Tool Glass").
First, human beings cooperatively utilize both of their hands to accomplish a wide variety of manual tasks, often with little or no accompanying cognitive effort. Doing so simply expedites those tasks, such as typing (where fingers of both hands are used in tandem to depress different keys on a keyboard) or manually writing (where one hand positions the paper and the other simultaneously manipulates a pen) or even tapping a nail into a piece of wood (where one hand holds the nail in place while the other lightly swings a hammer to hit a head of the nail), over what would otherwise be required to accomplish those tasks through use of a single hand. Alternatively, other tasks (such as mounting a spare tire on an automobile or handling another bulky object) could not be readily performed at all but for the use of two hands (or at least a suitable physical substitute for one hand).
As early as the mid-1980s, the art of computer interfaces teaches that in accomplishing a compound task, a one-handed computer interface is generally inferior to use of a two-handed interface, and particularly such a two-handed interface which splits the task into sub-tasks that, in turn, could be performed by a user through parallel and coordinated movement of both of his(her) hands. In that regard, see W. Buxton et al, "A Study in Two-Handed Input", Proceedings of CHI' 86, Boston, Mass., Apr. 13-17, 1986, pages 321-326 (hereinafter the "Buxton" paper). The Buxton paper teaches the use of an experimental two-handed user interface in which a preferred (dominant) hand (e.g., a right hand for a right-handed person) manipulates, in an absolute positioning mode, one input device, here a moveable digitizer (commonly referred to as a "puck") across a graphics tablet, while a non-preferred (non-dominant) hand (e.g., a left hand for the same person) simultaneously manipulates a second input device, here a so-called "slider". The slider allows one-dimensional input with an input amount being proportional to an amount through which a user moves a track on the slider up or down. Once various test subjects were trained to use these devices, the author of the Buxton paper observed that, through coordinated movement of both devices in parallel, users were able to markedly reduce the time needed to perform various compound user interface tasks, such as a combined selection/positioning task. This, of course, assumes that the two-handed interface is one that requires very little or no cognitive effort on the part of the user to employ. If the cognitive effort to use a given two-handed interface were to appreciably increase, then, of course, the utility of that particular interface, over a conventional one-handed interface, could become rather questionable.
Hence, as the Buxton paper recognizes, an additional degree of freedom afforded by a two-handed interface can potentially simplify a syntax of user input actions needed to accomplish a task, as compared to those needed through a conventional one-handed mouse-keyboard based interface to accomplish that same task, and thus result in an interface that is relatively simple and easy to use.
With this in mind, the art teaches the utility of coupling a Tool Glass to a two-handed UI. See, e.g., G. Kurtenbach et al "The Design of a GUI Paradigm based on Tablets, Two-hands and Transparency", Proceedings of Computer-Human Interaction (CHI) 1997, Atlanta, Ga., Mar. 22-27, 1997, pages 35-42; E. A. Bier et al, "A Taxonomy of See-through Tools", Proceedings of CHI 1994 Conference, April 1994, Boston, Mass., pages 517-523; and E. A. Bier et al "Toolglass and Magic Lenses: The See-Through Interface", Proceedings of the Annual Conference on Computer Graphics SIGGRAPH 93, Anaheim, Calif., Aug. 1-6, 1993, pages 73-80. In essence, a Tool Glass is a mobile semi-transparent sheet that contains a predefined group of semi-transparent icons (or widgets) and is overlaid on an application area of a display between an application display and a traditional cursor. Through a two-handed UI, each hand of a user can simultaneously manipulate a different input pointing device. The Tool Glass itself moves as an entity in concert with movement of the pointing device held by the non-preferred hand (which is a left hand for right-handed individuals). Movement of a cursor tracks movement of the other pointing device then being held by the preferred hand. Through coordinated hand motion, a user can easily position the Tool Glass with his(her) non-preferred hand such that an icon representing a desired operation is positioned over a particular object (being an operand) depicted in the application area and with his(her) preferred hand position the cursor over that icon and then click with (depress a button on) that device to invoke that operation. To facilitate understanding, I will differentiate between a displayed icon representing an operation and such an icon situated on a Tool Glass sheet by referring to the latter as a "tool". By, in effect, "clicking through" the tool, the operation associated with that tool is then executed on the underlying object.
By bringing a Tool Glass to a displayed object, rather than moving a cursor back and forth between the object and a peripheral toolbar, use of a Tool Glass advantageously reduces user distraction and a need for long mouse movements, and reduces the number of clicks required to perform various operations and hence the traditional cognitive effort previously associated with doing so. This, in turn, advantageously decreases user effort and task time, and increases user efficiency. Moreover, use of a Tool Glass advantageously increases available screen area permitted for useful application display by eliminating a need to allocate peripheral screen area for just displaying widgets.
Unfortunately, a Tool Glass, by virtue of its semi-transparent nature, obscures, to a certain degree, its underlying objects and adds clutter to an application area display. The user tolerates the clutter and obscuration as long as (s)he currently intends to use the Tool Glass. Otherwise, these effects are annoying.
Various conventional approaches exist to remedy these drawbacks--though these approaches have all proven inadequate. First, the user could explicitly dismiss a displayed Tool Glass sheet such as by clicking on, e.g., either a "close" button on the Tool Glass itself or a "exit" box in a pull-down menu. A user could also position a Tool Glass, through movement of his(her) non-preferred hand, at an off-screen location whenever it is not in use and then move (e.g., drag) that Tool Glass onto the screen to use it. Unfortunately and in either case, requiring a user to explicitly act in this fashion to either dismiss a Tool Glass and/or display it increases his(her) cognitive burden, and is thus best avoided. Alternatively, a Tool Glass UI could be implemented such that a Tool Glass is automatically dismissed as soon as its corresponding operation is invoked. However, with this approach, a Tool Glass may well be dismissed at an inopportune time, thus potentially frustrating the user.
Given the advantages attainable through use of a Tool Glass particularly when it is employed in conjunction with a two-handed interface, a need exists in the art for such a Tool Glass UI which can display or dismiss a Tool Glass at proper times, based on user activity. Doing so would eliminate unnecessary clutter and obscuration from a displayed application area. Furthermore, the display and dismissal should occur on an on-demand basis and impose only minimal, if any, cognitive burden on the user. If such an interface could be devised which overcomes the limitations inherent in conventional Tool Glass UIs, I would expect that the ensuing UI, once adopted in desktop user environments, should greatly enhance the "user experience".